This is yet another reason we cannot assume that data is 'objective' because it is quantitative in nature. For example, when constructing an experiment "an extreme groups design (e.g., assigning participants to high or low conditions) maximizes the variances of the components of the product term, it also results in much more power with respect to the interaction effect than would the corresponding observational design" (Cortina 2002: 343). Conversely, doing an experiment 'in the field' is likely to yield a less statistically-significant impact because of the inability to control the extremity of the variables. A recent study of the statistical power of research in the social sciences revealed that only 40% of all MIS studies had adequate statistical power to ensure that the probability that the null hypothesis would be rejected correctly at all times (Baroudi & Orlikowski 1989: 87). Significance criteria, sample estimate, and effect size, can all influence statistical power and once again, when dealing with human subjects, many additional variables can affect statistical power (Baroudi & Orlikowski 1989: 87). The use of certain statistical conventions can also yield inaccurate results, if deployed in an inappropriate fashion. For example, disregarding 'outliers' or extremes that impact the findings is a common practice and may be appropriate or inappropriate, depending upon the circumstances, as can filling in missing results to enable the statistical analysis to be done in the first place (Gardenier & Resnik 2002: 68). If the outliers are not genuine 'outliers' that can be explained convincingly as such or the missing data cannot be extrapolated easily, it can produce wildly inaccurate results.

Sometimes the misuse of quantitative data is unintentional, other times it is deliberate. In some instances, data may be obtained in a fraudulent and unethical manner, deliberately designed to produce a particular, false result (such as editing, cleaning, or mining data) or aspects of...

Misuse is commonly divided into two categories: falsification, in which real data is manipulated and fabrication, in which data is literally made up out of 'whole cloth' (Gardenier & Resnik 2002: 70). Obviously, false or fabricated data is clearly 'bad research,' given that is useless in terms of proving or disproving the initial hypothesis of the research and, in a worst case scenario, can impact people's lives causing needless harm.
Researchers who are over-eager to prove a hypothesis they are sure is 'correct' or who have a financial or career-related interest in a particular result (such as proving that a particular drug is effective when working for a drug company) may engage in such manipulation, motivated by egotism or bias. A lack of adequate education can likewise yield problems. Overreliance upon computers vs. intelligent hand-checking and acceptance of various conventions within the discipline with arbitrary assigned values can likewise compound the problem (Gardenier & Resnik 2002: 71).

All of this does not mean that we should throw up our hands and abandon statistical research. However, it is important counsel that the appearance of numbers is no guarantee that the research in question is of better quality and more accurate and generalizable than a small qualitative study. The cliche that 'numbers don't lie' is untrue, given that numbers are always accumulated by human intelligence and via human-created designs.

References

Baroudi, J. & Orlikowski, W. (1989). The problem of statistical power in MIS Research.

MIS Quarterly, 13 (1): 87-106

Cortina, J.M. (2002). Big things have small beginnings: An assortment of 'minor'

methodological misunderstandings. Journal of Management, 28(3), 339-362.

Gardenier, J. & Resnik, D. (2002). The…